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William  F.  Kramer,  Mehmet  Sahinoglu,  and  David  Ang 


This  research  article  aims  to  identify  and  introduce  cost-saving 
measures  for  increasing  the  return  on  investment  during  the  Software 
Development  Life  Cycle  (SDLC)  through  selected  quantitative  analyses 
employing  both  the  Monte  Carlo  Simulation  and  Discrete  Event  Simu¬ 
lation  approaches.  Through  the  use  of  modeling  and  simulation,  the 
authors  develop  quantitative  analysis  for  discovering  financial  cost 
and  impact  when  meeting  future  demands  of  an  organization’s  SDLC 
management  process  associated  with  error  rates.  Though  this  sounds 
like  an  easy  and  open  practice,  it  is  uncommon  for  most  competitors  to 
provide  empirical  data  outlining  their  error  rates  associated  with  each 
of  the  SDLC  phases  nor  do  they  normally  disclose  the  impact  of  such 
error  rates  on  the  overall  development  effort.  The  approach  presented 
in  this  article  is  more  plausible  and  scientific  than  the  conventional 
wait-and-see,  whatever-fate-may-bring  approach  with  its  accompanying 
unpleasant  surprises,  often  resulting  in  wasted  resources  and  time. 


Keywords:  discrete  event  simulation  (DES),  Monte  Carlo  simulation  (MCS),  error  or 
defect  rate,  return  on  investment  (ROI),  software  development  life  cycle  (SDLC) 
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The  science  behind  software  development  in  metric  terms  of  return 
on  investment  (ROI)  is  well  known  and  taught  by  many.  Much  work  has 
been  accomplished  in  this  area  albeit  lacking  details  of  execution  on  a 
real-life  problem  (Ferreira,  Collofello,  Shunk,  &  Mackulak,  2009;  Zhang, 
Kitchenham,  &  Pfahl,  2008;  Zhang,  Kitchenham,  &  Pfahl,  2010).  The  art 
of  software  development  is  a  learned  behavior  and  not  one  with  which 
everyone  becomes  comfortable  due  to  its  intricacies  and  learning  cycle. 
The  same  may  be  said  with  respect  to  software  development  life  cycle 
(SDLC)  management  and  distribution  as  depicted  in  Table  1,  where  the 
different  phases  of  an  SDLC  process,  when  applied,  provide  specific 
inputs  and  expected  outputs. 


TABLE  1.  LIFE  CYCLE  PHASE  FLOW 
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Life  Cycle  Phase  (Process)  Flow 


As  with  many  processes,  there  is  a  beginning  point  and  a  delivery 
epoch.  SDLC  methodology  is  no  different.  It  enables  standardization 
for  planning  and  organizing,  and  facilitates  cost  estimation.  Though 
there  are  several  different  models  available,  many  are  tweaked  to  best 
fit  the  current  process  or  a  sequence  of  activities  in  a  software  develop¬ 
ment  project.  The  life  cycle  used  in  this  article  (Table  1)  has  nine  phases 
beginning  with  the  requirement  review  and  ending  with  the  deploy¬ 
ment  decision.  As  one  begins  with  the  first  phase  (i.e.,  requirement 
review)  and  moves  right,  software  developers  will  observe,  at  a  mini¬ 
mum,  the  activities  that  must  be  performed  in  the  phase  (keep  in  mind 
this  is  a  high-level  depiction).  Moving  right,  there  is  a  decision  to  be 
made  whether  to 
proceed  to  the  next 
phase  or  recycle 
back  through  the 
current  phase  for 
further  refinement. 


This  decision  is 
only  one  of  many 
for  the  phases; 
however,  it  might 
be  the  most  cru¬ 
cial.  Not  only  will 
schedule  and  cost 
be  impacted,  but 
phase  errors  will 
drive  substantial 
cost  as  well.  An 
organization  needs 
to  understand  the 

impact,  and  that  is  the  intent  of  this  article— namely  to  show  the  phase 
error  impact  to  the  SDLC,  thereby  reducing  overall  project  manage¬ 
ment  cost  by  improving  the  error  rate. 


Each  phase  will  generate  its  own  success  criteria,  allowing  a  develop¬ 
ment  team  to  anticipate  the  degree  of  success  that  can  be  expected 
throughout  the  life  cycle.  Unfortunately,  as  a  development  team  moves 
through  the  SDLC  process,  it  is  common  to  shift  expected  outputs  to 
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the  right  and  ultimately  into  the  next  phase,  if  only  to  remain  on  track 
regarding  the  end  schedule  or  an  expected  financial  burn  rate.  Ultimately, 
reality  will  set  in  and  a  price  to  be  paid  will  become  readily  apparent, 
whether  it  be  in  the  form  of  a  scheduling  or  financial  disruption. 


This  may  be  even  more  prevalent  when  it  comes  to 
*he  acquisition  of  custom  software.  To  be  bet¬ 
ter  prepared  for  the  impact  of  the  shifting 
deliverables  associated  with  the  SDLC  man¬ 
agement  process,  one  must  understand  the 
intricacies  of  the  process  and  especially  the 


impacts  associated  with  a  product  that 
.  is  either  late  or  overbudgeted.  Using 
a  discrete  event  simulation  (DES) 
and  Monte  Carlo  simulation  (MCS), 


as  combined,  may  assist  in  quantifying  a 


scenario  impact.  The  primary  raison  d’etre 


of  this  article  is  to  demonstrate  the  poten¬ 


tial  for  modeling  the  SDLC  management 
process  and  bring  the  cost-saving  factor 
forward  to  improve  the  ROI  by  employ¬ 
ing  statistical  simulation  techniques. 


Therefore,  the  basis  of  this  article  is  to  bring 
attention  to  the  use  of  modeling  and  simulation 
(M&S)  in  developing  a  quantitative  analysis 
for  discovering  potential  scheduling  and  financial 
;  within  the  parameters  of  meeting  future  demands 
of  an  organization’s  SDLC  management  process. 
More  specifically,  the  potential  impact  is  asso¬ 


ciated  with  errors  accruing  and  accumulating 


throughout  the  process.  That  being  said,  one 
wjy  must  be  mindful  that  the  methods  used  to  corn¬ 
er  pile  this  research  article  rely  equally  upon  the  art  of 
simulation  as  well  as  the  ever-enduring  statistical  and 
mathematical  sciences  behind  the  art  of  simulation.  The 


statistical  and  mathematical  computations  used  a  significant  amount  of 
data  gleaned  from  many  years  of  software  development  experience.  It  is, 
however,  through  these  years  of  experience  with  software  development 
projects  that  we  have  come  to  appreciate  an  SDLC  management  process. 
Likewise,  it  is  also  during  this  process  that  we  have  learned  to  exercise 
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a  degree  of  caution  when  evaluating  those  bidding  to  perform  custom 
software  development,  who  typically  bid  on  the  process  with  some  degree 
of  naivete  that  views  every  requirement,  algorithm,  and  interface  as  a 
nonissue  and  the  work  as  always  straightforward.  Most  of  the  time,  this 
is  not  true,  since  hiccups  invariably  surface  along  the  way,  whether  in 
the  form  of  undefined  requirements  or  bad  test  data.  More  often  than 
not,  unforeseen  events  occur,  which  ultimately  impact  both  schedule 
and  cost  to  the  users’  disadvantage. 


Modeling  and  Simulation 
Methodology  for  a  Case  Study 

To  identify  and  incorporate  software  life-cycle  phases  along  with 
function  point  analysis,  software  managers  ought  to  associate  the  error 
rate  per  phase  with  the  time  distribution  per  phase.  Organizations  per¬ 
forming  standard  unit,  integration,  and  functional  testing  will  likely 
only  remove  approximately  70%  of  defects  during  the  life-cycle  phases 
(Jones,  2008).  This  practice  will  allow  other  defects  to  run  through 
the  life  cycle  until  the  bottleneck  becomes  apparent  in  the  final  test¬ 
ing  phase.  The  model  introduction  takes  this  into  account  and  assists 
with  providing  a  rough  order  of  magnitude  (ROM)  to  the  level  of  effort 
a  program  may  encounter.  In  addition,  the  model  also  provides  an  alter¬ 
native  approach  to  facilitate  ROMs  with  the  appropriate  schedule  and 
additional  resources. 

Computer  M&S,  as  programs  or  networks  of  computers  mimicking  the 
execution  of  an  abstract  model  of  many  natural  systems  from  physi¬ 
cal  and  life  sciences  to  social  and  managerial  sciences,  and  primarily 
engineering,  have  become  an  integral  part  of  digital  experimentation. 
M&S  proves  useful  to  estimate  the  performance  of  complex  engineering 
systems  when  too  prohibitive  for  analytical  solutions.  A  simulation  is 
defined  as  the  reproduction  of  an  event  with  the  use  of  scientific  models. 
A  model  is  a  physical,  mathematical,  or  other  logical  representation  of 
a  system,  process,  or  phenomenon.  Time-independent  static  MGS  and, 
conversely,  dynamic  DES  (to  manage  events  in  real  time  for  engineer¬ 
ing  applications)  have  been  extensively  reviewed  (Sahinoglu,  2013). 
Taxonomy-wise,  simulated  computer  models  maybe  stochastic  or  deter¬ 
ministic,  and  dynamic  or  static,  and  discrete  or  continuous.  Computer 
simulation  has  been  widely  used  in  engineering  systems  to  validate  the 
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effectiveness  of  tentative  decisions  regarding  a  new  plan  or  schedule,  or 
its  outcomes,  without  experiencing  the  actual  conditions,  which  could 
cost  more  resources  or  partial  to  full  destruction  such  as  in  the  simula¬ 
tion  of  the  nuclear  bomb  (Sahinoglu,  2007).  In  a  book  titled  Simulation 
Engineering  by  Jim  Ledin  (2001),  the  author  outlines  his  twofold  purpose 
as  follows: 

i)  Simulation  is  an  approach  that  can  significantly  accelerate 
the  product  development  cycle  and  provide  higher  quality  in  the 
final  system. 

ii)  A  simulation  contains  a  set  of  mathematical  models  of  one 
or  more  dynamic  systems  and  the  interactions  between  those 
systems  and  their  environment,  (p.  1) 

Moreover,  the  Institute  for  Electrical  and  Electronics  Engineers' 
Spectrum  (June  2012)  emphasized  that  the  M&S  effect  is  a  creative 
and  time-saving  topic  of  interest  relevant  to  automotive  engineering  of 
hybrid  vehicles,  finding  solutions  to  treating  nuclear  waste,  upgrading  the 
nuts  and  bolts  of  the  electrical  power  (Smart)  grid,  and  supercomputing 
research,  among  other  areas  (Aoyama,  2012). 

Simulation  Approach 

Table  2  depicts  the  conduct  of  an  error  rate  analysis  within  the 
parameters  of  the  SDLC  management  process.  To  better  depict  the  prob¬ 
ability  distribution.  Table  2  associates  the  probability  distributions  with 
each  phase  of  the  life  cycle.  Keeping  in  mind  a  waterfall  model  is  in  play, 
future  research  may  requre  further  phase  delineation  among  the  many 
attributes  of  the  phases.  Note  that: 

•  There  is  a  need  to  simulate  and  model  error  rates  within  the 
SDLC  process.  Schedules  and  costs  are  impacted. 

•  Many  models,  such  as  waterfall,  Agile,  SCRUM,  RAD,  time- 
box,  and  spiral  development  methodologies  exist  today  and 
could  be  used  (Zhang,  et  al.,  2010). 

•  This  simulation  model  (Table  2)  focuses  on  the  error  rates 
associated  with  waterfall  methodologies. 


180 


Defense  ARJ,  April  201S,  Vol.  22  No.  2 : 174-191 


April  2015 


•  In  order  to  determine  cost  per  cycle  and  average  cost  per 
phase  when  using  a  development  rate  consisting  of  function 
point  per  staff  month,  calculate  the  error  rates  per  phase 
and  then  aggregate  with  the  suggested  cost  model. 


Algorithmic  Step-by-Step  Approach  Using  Statistical 
Random  Number  Generation 

Table  3  depicts  iterations  1-1,000  and  provides  the  details/samplings 
used  in  the  simulation  correlating  the  phases  with  probability  distri¬ 
butions,  the  defect  rates,  repair  rates,  lambda,  mu,  standard  deviation 
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(STD),  and  mean  (Malone  &  Mizell,  2009).  The  average  of  the  sampling 
was  used  along  with  a  180-day  SDLC  to  determine  defect  rates  per  phase. 
These  were  used  in  the  Java  application  to  simulate  and  provide  input  to 
the  findings  in  Table  4.  Note  the  following; 
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•  Function  point  count  is  maintained  at  one  function  point  for 
the  life-cycle  period  of  180  days.  With  the  distribution  per 
phase  identified  along  with  the  days  per  phase,  the  Average 
Defects  per  Phase  (ADP)  is  introduced  with  the  summation 
of  the  ADP  to  be  the  average  defect  per  one  function  point. 

•  Next,  the  Average  Defects  per  Day  (ADD)  is  calculated 
by  dividing  the  ADP  by  the  Days  per  Phase.  This  output 
becomes  our  lambda  (A,)  in  the  phase  calculation  in  deter¬ 
mining  our  Probability  of  Waiting  (PoW). 

•  The  Average  Repairs  per  Day  (ARD)  is  determined  by  mul¬ 
tiplying  the  ADD  by  our  utilization  factor  of  a  constant  0.8 
(80  percent)  from  best  practices  (Malone  &  Mizell,  2009). 

This  output  becomes  our  mu  (jt),  also  used  in  determining 
the  PoW. 

Results 

Factors  used  to  obtain  results  (Table  3)  follow; 

•  Average  Defects  per  Phase  =  (summation  of  each  phase 
distribution)/iterations 

•  Days  per  Phase  =  variable  set  by  experience 

•  Average  Defects  per  Day  =  (Average  Defects  per  Phase)/ 
(Days  per  Phase) 

•  Average  Repairs  per  Day  =  (Average  Defects  per  Day)/utili- 
zation  factor. 

To  make  use  of  the  facts  in  Table  3,  a  Java  application  (see  Appendix, 
Java  Source  Code  First  Page)  was  developed  to  conduct  several  thousand 
runs  for  the  simulation  and  ultimately  provide  a  statistical  summary  to 
support  Excel  findings.  The  facts  from  the  spreadsheet  shown  in  Table 
4  were  placed  into  this  homebrewed  java  application  where  the  user  can 
identify  the  inputs,  the  number  of  runs,  and  lastly,  can  run  with  either  a 
single-team  or  a  two-team  simulation. 

Table  4  represents  only  one  screen  shot  with  a  single  distribution,  while 
arbitrarily  using  cost  per  hour  of  $55,  team  size  of  10,  and  work  hours  per 
day  to  equal  8.  One  can  vary  the  cost  factors.  Taking  these  factors  into 
account,  the  cost  formula  in  Equation  (1)  is  as  follows: 
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Total  Cost  =  (Days  to  repair  •  work  hours  per  day  •  team 
size  •  hourly  rate).  (1) 

We  can  begin  to  readily  determine  that  the  errors  per  phase  quickly 
outpace  the  efforts  of  a  single  developer  and  throw  the  schedule  and  cost 
model  far  to  the  right.  However,  by  adding  a  second  development  team  to 
assist  with  the  fixing  of  the  errors  per  phase,  the  cost  and  schedule  are 
only  slightly  impacted  (Malone  &  Mizell,  2009). 

One  can  better  appreciate  the  long-term  impacts  when  dealing  with 
contracts  and  why  the  lower  bid  may  initially  seem  the  best  value;  how¬ 
ever,  with  the  software  development  life  cycle,  this  may  not  be  the  case. 
Improper  preliminary  analysis  and  use  of  resources  could  easily  whirl 
the  schedule  and  cost  into  an  embarrassing  tailspin.  The  core  of  this 
research  precludes  this  handicap. 

Other  findings  and  Excel  spreadsheet  results  highlighted  in  Table  4 
follow. 


•  PoW  is  multiplied  with  the  Days  per  Phase  to  obtain  the 
Days  to  Repair  for  each  specific  team. 

•  Multiple  variables  are  added  to  obtain  realistic  cost  of  soft¬ 
ware  development  teams  (such  as  hourly  rate  of  developers, 
team  size,  and  hours  per  workday). 

•  The  formula  used  for  each  team  is:  Total  Cost  =  (repair  days 
•  work  hours  •  team  size  •  hourly  cost). 

Validation 

Does  the  lowest  dollar  contract  actually  deliver  the  best  value?  This 
is  what  the  research  confirms  positively. 

Verification 

Validation  of  error  rates  and  function  point  rates  came  from  Jones 
(2008). 

Outcomes 

Development  teams  can  determine  cost  at  granular  phases  within 
the  SDLC  as  it  pertains  to  error  rates  within  software  development. 
Upon  running  the  simulation,  the  aggregated  results  show  significant 
financial  benefits.  Factors  used  to  obtain  results  are  shown  in  Table  3 
(Malone  &  Mizell,  2009). 
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Conclusions 

The  article  responds  to  the  following  question:  When  required  to 
analyze  best-value  contracts  without  using  a  simulation  model,  does 
the  requestor  actually  obtain  true  cost  by  analyzing  a  single  entity 
to  develop  software  versus  aggregated  cost  (Table  4)  delivered  from 
an  additional  pool  of  resources?  Future  work  along  with  inputs  from 
aoftwaro  development  ooot  modolo  will  go  a  long  way  in  producing  a 

If  the  errors  are  identified  in  the  early  stages  of  a 
software  development  acquisition,  contracting 
officers  may  be  in  a  better  position  to  avoid  the 
lowest  contract  bid  if  they  understand  where 
proper  resources,  when  applied,  may  actually 
decrease  cost  and  schedule,  thus  delivering  a 
successful  acquisition  and  software  functionality. 


better  understanding  of  the  true  cost  of  software  development  and  why 
there  seems  to  be  a  schedule  shift  as  the  SDLC  runs  through  its  phases. 
This  project  scratches  the  surface  by  showing  that  the  assumption  by 
most  software  developers  that  all  contracts  and  estimates  provided  are 
realistic,  does  not  really  portray  the  impact  of  errors  to  the  schedules, 
which  further  increases  cost.  Some  conclusive  findings  of  interest  are 
outlined  below: 

•  Average  cost  per  phase  with  single  team  to  fix  errors  is  an 
estimated  $628,421.20  with  the  original  summation  of  180 
days  per  phase. 

•  Adding  an  additional  team  to  focus  on  errors,  thereby 
increasing  the  cost  for  labor  for  two  teams,  equates  to 
42.14%  savings.  This  is  readily  discerned  in  the  reduced 
number  of  days  to  fix  the  errors.  In  fact,  the  second  team 
will  cost  an  estimated  $363,633.60  in  labor.  The  overall 
estimated  savings  is  $264,787.60  for  the  cost  of  the  repairs. 

•  Future  and  long-term  analysis  should  focus  on  specific 
methodologies  as  well  as  on  the  coding  language. 
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•  Many  organizations  have  invested  in  the  use  of  the  waterfall 
methodology  and  have  been  slow  to  appreciate  the  potential 
cost  and  schedule  impact  from  error  rates  within  the  mul¬ 
tiple  phases  of  the  SDLC. 

•  This  article  aimed  to  present  a  DES  and  MGS  to  determine 
an  outcome  that  can  be  used  to  improve  a  process  and  cut 
costs.  The  error  rate  analysis  project  has  done  just  that. 

•  Through  lengthy  discussions  about  rates  within  the  MGS 
portion  and  the  impact  on  business  development  systems, 
additional  research  and  refinement  may  be  sought  to  fur¬ 
ther  develop  the  phase  rates  from  within  an  organization. 
Additional  research  will  provide  better  understanding  of 
the  impact  for  long-term  software  development  and  error 
rate  impact. 

•  It  is  hoped  that  this  and  later  work  will  enable  future  profes¬ 
sionals  in  software  development  acquisition  to  establish  a 
more  definite  cost  analysis  when  confronting  quantifiable 
data  such  as  function  points  and  development  languages  to 
give  them  a  better  understanding  of  the  impact  of  develop¬ 
ment  errors  within  the  different  phases  of  the  waterfall 
SDLG. 

An  SDLG  is  a  methodological  process  that  from  a  high  level  can  be  used 
to  determine  schedules  and  costs  and  identify  bottlenecks.  However,  it 
seems  only  recently  that  declining  information  technology  budgets  and 
increasing  delivery  costs  require  us  to  slice  the  life  cycle  into  further 
granularity  to  understand  better  the  cost  and  schedule  impacts.  In  an 
attempt  to  correlate  errors  with  phases  and  cost  to  fix,  a  prevailing 
assumption  is  that  the  cost  of  errors  is  fiat.  However,  this  may  not  be  so. 
If  the  errors  are  identified  in  the  early  stages  of  a  software  development 
acquisition,  contracting  officers  may  be  in  a  better  position  to  avoid  the 
lowest  contract  bid  if  they  understand  where  proper  resources,  when 
applied,  may  actually  decrease  cost  and  schedule,  thus  delivering  a  suc¬ 
cessful  acquisition  and  software  functionality. 
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Appendix 


JAVA  SOURCE  CODE  FIRST  PAGE 


//package  negexp; 

//  W.  Kramer 

import  java.awt.*; 
import  java.awt. event.*; 
import  javax. swing.*; 
import  java. util. Random; 
import  java. text.*; 

public  class  NegExp  extends  JFrame  { 

//elements  of  user  interface 
private  JLabel  triaIsJLabel; 
private  JLabel  meanJLabel; 
private  JLabel  devJLabel; 
private  JLabel  MuJLabel; 
private  JLabel  BetaServiceJLabel; 
private  JLabel  errorRateJLabel; 
private  JLabel  LambdaJLabel; 
private  JLabel  BetaJLabel; 

private  JLabel  servTimeJLabel;  //package  negexp; 
//  W.  Kramer 

import  java.awt.*; 
import  java.awt. event.*; 
import  javax. swing.*; 
import  java. util. Random; 
import  java. text.*; 

public  class  NegExp  extends  JFrame  { 
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